Parallelized Stochastic Gradient Descent

With the increase in available data parallel machine learning has become an in-creasingly pressing problem. In this paper we present the first parallel stochasticgradient descent algorithm including a detailed analysis and experimental evi-dence. Unlike prior work on parallel optimization algorithms [5, 7] our variantcomes with parallel acceleration guarantees and it poses no overly tight latencyconstraints, which might only be available in the multicore setting. Our analy-sis introduces a novel proof technique — contractive mappings to quantify thespeed of convergence of parameter distributions to their asymptotic limits. As aside effect this answers the question of how quickly stochastic gradient descentalgorithms reach the asymptotically normal regime [1, 8].